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LONG-TERM  GOALS 

My  long  term  goal  is  to  understand  the  fundamental  limitations  on  short-tenn,  mesoscale  predictability. 
Of  particular  interest  are  the  effects  of  mid-scale  turbulence  on  episodic  initiations,  or  triggering 
mechanisms,  and  the  sensitivities  of  the  predictions  of  consequential  events  to  these  mechanisms.  The 
issue  is  to  develop  techniques  by  which  we  can  objectively  distinguish  between  hard  and  impossible 
prediction  problems.  We  are  especially  interested  in  applying  these  results  to  provide  guidance  for  the 
design  and  interpretation  of  the  ensemble  forecast  systems. 

OBJECTIVES 

I  wish  to  investigate  the  concept  of  a  well-posed  prediction  problem,  a  prediction  problem  that  has  a 
theoretical  solution.  The  complementary  set  of  ill-posed  prediction  problems  are  those  for  whom  no 
prediction  algorithm  or  model  can  be  developed,  which  will  satisfy  the  required  forecast  accuracy  or 
skill.  A  prediction  problem  may  be  well-pose,  even  though  the  prediction  technology  has  not  yet  been 
developed.  Thus  the  well-posed  concept  separates  the  impossible  predition  problems  from  those  that 
are  just  very  challanging.  Studying  well-posed  prediction  problems  includes  quantifying  the 
relationship  between  the  prediction  goals  and  the  verification  results. 

These  investigations  also  involve  considerations  of  the  methodology  for  quantifying  the  skill  of 
prediction  systems.  Since  the  specification  of  a  well-posed  problem  includes  the  selection  of  an 
objective  verification  procedure,  it  is  necessary  to  have  a  sound  statistical  foundation  for  verification. 
We  examine  how  this  paradigm  applies  to  a  simple  classical  chaotic  system  and  predictions  of  states  on 
its  attractor.  Studies  of  the  predictability  of  the  Logistic  Mapping  from  various  initial  states  on  its 
attractor  provide  insight.  The  notion  of  condition  entropy  is  introduced  as  the  skill  metric  for  these 
studies. 

APPROACH 

Traditional  statistical  measures  of  predictability  of  chaotic  systems,  such  as  its  invariant  measure  and 
entropy,  measure  the  climatology  of  the  attractor.  Conditional  measures  relate  to  predictability  for 
specific  initial  states.  We  develop  techniques  for  computing  the  conditional  probabilities  and  the 
associated  conditional  entropy.  The  conditional  entropy  estimates  the  number  of  lost  bits  of 
information  between  the  initial  condition  and  the  prediction.  Thus  if  a  forecast  from  the  initial 
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condition  x  is  initiated  with  8-bit  precision,  and  the  forecast  has  conditional  entropy  equal  to  2,  then  the 
prediction  is  expected  to  have  6-bit  accuracy. 


We  also  investigate  the  statistical  estimation  of  skill  metrics  for  categorical  forecasts.  These  metrics  are 
candidates  for  measuring  the  skill  of  traditional  weather  forecast  systems,  and  sample  estimations  are 
useful  only  if  they  are  repeatable,  that  is,  they  do  not  depend  on  weather  frequency  during  a  verification 
exercise.  In  the  course  of  these  studies,  we  also  determine  the  relationship  of  skill  statistics  to 
ecconomic  benefits. 

WORK  COMPLETED 

We  have  completed  two  studies  this  year: 

1 .  An  extensive  study  of  skill  statistics  for  categorical  forecast  systems  has  been  completed.  Both 
detenninistic  and  probabilistic  forecasts  are  investigated. 

2.  The  notion  of  conditional  entropy  has  been  developed,  together  with  an  estimation  procedure.  This 
measure  has  been  applied  to  several  cases  of  the  Logistic  Mapping  and  it  has  been  found  that  there 
are  cases  where  the  traditional  predictability  measures  are  unduly  pessimistic. 

RESULTS 

Skill  statistics  for  deterministic  binary  forecasts  are  investigated  from  the  viewpoints  of  repeatability, 
and  relevance.  These  statistics  are  discussed  from  the  viewpoints  of  conditional  probabilities,  their 
estimation,  and  their  relationships  to  Type  I  and  Type  II  forecast  skill.  Most  of  the  classic  skill  statistics 
for  detenninistic  forecasts  are  relevant,  but  the  TSS  is  the  only  repeatable  classic  skill  statistic. 

The  skill  statistics  are  interpreted  in  terms  of  the  expected  benefits  of  applications;  two  variants  of  the 
TSS  are  found  to  be  important. 

The  worth  skill  statistic  WSSu  is  fonned  by  rescaling  TSS,  based  on  the  application  use  factor  U, 
which  is  defined  in  tenns  of  the  user's  gain-loss  factors.  WSSu  is  proportional  to  the  benefits  recovered 
and  to  the  benefits-per-trial  for  the  class  of  users  with  use  factor  U. 

The  worth  ratio  WR  is  the  ratio  of  the  tenns  of  TSS.  Tuning  a  forecast  system  to  maximize  WR  will 
produce  nearly  maximum  values  for  each  WSSu,  and  will  provide  maximum  benefits-per-action  for  all 
users.  WR  is  a  valuational  skill  statistic;  whenever  there  is  a  change  to  a  forecast  system  that  increases 
WR,  then  there  is  a  conesponding  increase  in  benefits-per-action  for  all  users.  The  WR  &  WSS  Graph 
is  introduced  as  tool  for  optimizing  a  forecast  system  that  depends  on  a  continuous  parameter.  It  is  an 
extension  of  the  Relative  Operating  Characteristic  Diagram  of  Mason. 
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Figure  1.  The  WR  &  WSS  Graph,  plotted  against  P(T\Z)  for  synthetic  data.  The  maximum  values  of 
each  WSSu  curve  are  marked  as  solid,  together  with  the  corresponding  point  on  p.  Note  the 
locations  of  WSSu  maxima  to  the  right  of  the  maximum  ofWR.  Recall  that  WSSi  =  TSS.  The 
function  p  represents  the  derivative  of  P(T\  W)  in  this  parameterization,  and  is  included  to  illustrate 

the  theoretical  relationships. 

A  context  is  developed  that  provides  extensions  of  these  skill  statistics  to  probabilistic  binary  forecasts. 
Since  the  extensions  are  defined  in  terms  of  the  conditional  probabilities,  the  notions  of  repeatability 
and  relevance  remain  applicable.  The  popular  Brier  Score  is  found  to  be  not  repeatable.  The  extensions 
of  TSS,  WSSu,  and  WR  provide  repeatable  and  relevant  measures  of  the  skill  probabilistic  forecasts. 
These  measures  also  have  the  property  that  they  have  a  consistent  interpretation  as  probabilistic 
forecasts  become  more  certain. 

In  order  to  determine  the  value  of  probability  forecasts  in  applications,  it  is  necessary  to  consider  the 
action  strategy  of  the  user.  Two  action  strategies  are  considered:  full  commitment  and  partial 
commitment.  The  benefit  analysis  for  detenninistic  forecasts  is  directly  applicable  to  the  full 
commitment  strategy  and  to  the  simple  partial  commitment  strategy.  We  show  that  the  optimal  partial 
commitment  strategy  is  a  full  commitment  strategy.  The  WR  &  WSS  Graph  provides  a  graphical 
process  for  determining  the  decision  threshold  that  optimizes  the  commitment  strategy  for  probabilistic 
forecasts. 


Our  second  class  of  investigations  relates  to  studies  of  the  predictability  of  chaotic  systems.  The 
analysis  of  these  systems  is  conducted  through  numerical  experiments.  A  key  factor  is  the 
predictability,  with  a  perfect  model,  when  there  is  slight  uncertainty  in  the  initial  condition.  The 
probability  distribution  of  the  resulting  forecasts  is  a  key  factor.  We  represent  the  density  of  this 
distribution  by  the  conditional  probability  p(y|x),  where  y  is  a  possible  forecast  and  x  is  the  initial 
condition  i  the  space  X.  Conditional  entropy  converts  this  condition  probability  to  a  measure  of  the 
expected  loss  of  resolution  in  the  forecast: 


Conditional  Entropy  of  y:  I(y|x)  =  -  log  p(y|x)  the  uncertainty  in  y|x 

Conditional  Entropy  of  forecasts:  I(Y|x)  =  J  p(y|x)  I(y|x)  dy  average  uncertainty  |x 

Conditional  Entropy  of  X:  I(Y|X)  =  JJ  p(x,y)  I(y|x)  dydx  =  J  p(x)  I(Y|x)  dx 

the  average  uncertainty  of  all  forecasts  on  X. 

We  compute  the  conditional  entropy  of  the  logistic  map  for  r  =  3.826,  a  location  where  the  Logistic 
Mapping  is  strongly  chaotic  and  has  entropy  10.3  for  12-bit  initial  conditions.  We  see  that  there  is  a 
substantial  variation  of  the  predictability  across  the  attractor,  that  forecasts  from  the  initial  conditions 
near  .5  remain  predictable  (Figure  1),  and  that  there  is  a  linear  degradation  of  the  predictability  with 
forecast  lead  time  (Figure  2).  We  make  no  special  claim  about  this  last  result,  even  though  we  have 
seen  it  repeatedly  as  we  have  varied  r.  It  may  be  a  property  of  the  Logistic  Mapping,  rather  than  a 
predictability  result. 


Figure  2.  Conditional  Entropy  I(Y\x)  as  a  function  of  x  for  perfect  forecasts  of  time  1,2,4,  and  10 
from  the  attractor  of  the  Logistic  Mapping  with  r=3.826. 
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Figure  3.  The  change  in  the  average  conditional  entropy  as  a  function  of forecast  lead  time  for 
perfect  forecasts  of  the  Logistic  Mapping  with  r=3.826.  Lambda  is  the  average  Lyapunov  exponent. 

IMPACT/APPLICATIONS 

The  development  of  skill  statistics  that  are  dependable  and  which  reflect  user  benefit  has  wide  potential 
application.  These  measures  of  skill  have  the  additional  merit  that  they  unify  the  scoring  of 
deterministic  and  probabilistic  forecasts. 

TRANSITIONS 

The  results  concerning  skill  statistics  are  being  considered  for  the  evaluation  of  the  skill  of  forecast 
products  in  the  FAA' s  Aviation  Weather  Research  Program. 

RELATED  PROJECTS 

The  skill  statistics  have  been  used  to  evaluate  the  forecast  skill  for  four  forecast  models  that  were 
developed  by  the  FAA’ s  Aviation  Weather  Research  Program  for  the  forecast  of  the  dissipation  of  low 
clouds  at  San  Francisco  International  Airport. 
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